Nonsmooth Optimization using Classification and Regression Trees

Nonsmooth local optimization problems occur in many fields, including engineering, mathematics, and economics. In economics, nonsmooth problems can be found in finance, mathematical economics and production theory. Examples include, nonsmooth utility maximization and exact penalty functions. However there are few convergent optimization algorithms to solve general nonsmooth or discontinuous problems. Random search methods can be applied to such problems because they do not require gradient information. However such methods search for global, rather than local, solutions and are often computationally expensive to use in practice. To apply random search methods to nonsmooth local minimization we employ techniques from classification theory, in particular classification and regression trees (CART). CART provides a way of partitioning an optimization region S into sub-regions. Imposing a classification on a set of random points {x : x ∈ S} with respect to function values allows a partition to be formed. Here we consider points are of two categories, either high or low. Hence, each sub-region of the partition is classified as either high or low. The sub-regions are defined by hyperrectangles because binary classification trees are used. A local minimization algorithm is introduced. The method is set up to solve nonsmooth or discontinuous problems in an n-dimensional box S. The algorithm alternates between a partition and sampling phase. Firstly the optimization region is partitioned with CART using a training data set T , identifying low and high subregions. A new batch of points is then distributed into the low regions with an increased probability distribution. The new points are added to the training data and the method repeats, until a stopping rule terminates the algorithm. We have found that the use of CART in nonsmooth optimization is very effective. The CART procedure itself in computationally cheap to evaluate and identifies low sub-regions of S well. The hyperrectangular sub-regions are easy to resample, allowing for our iterative resampling algorithm. This method is applicable to nonsmooth and discontinuous objective functions.